Approximation theory of output statistics

Given a channel and an input process we study the minimum randomness of those input processes whose output statistics approximate the original output statistics with arbitrary accuracy. We introduce the notion of resolvability of a channel, defined as the number of random bits required per channel use in order to generate an input that achieves arbitrarily accurate approximation of the output statistics for any given input process. We obtain a general formula for resolvability which holds regardless of the channel memory structure. We show that, for most channels, resolvability is equal to Shannon capacity. By-products of our analysis are a general formula for the minimum achievable (jixed-length) source coding rate of any finite-alphabet source, and a strong converse of the identijication coding theorem, which holds for any channel that satisfies the strong converse of the channel coding theorem. There are situations of practical interest where a random process needs to be generated with some specified statistics. In order to generate a random process we assume that a primary random source with an equiprobable distribution is available (e.g. a stream of independent fair coin flips). A key measure of the complexity of a random process is the rate at which its most efficient generator requires random bits, in order to generate every samplepath of the random process. This question becomes particularly interesting when rather than requiring the exact reproduction of the desired statistics, we require an arbitrarily accurate approximation of the finite-dimensional distributions. This requires the introduction of a measure of distance between the desired and generated distributions; in t h ~ s paper we focus most of our attention on the variational or 1 , distance. We prove that for any random process the minimum complexity required to approximate its statistics is equal to its minimum achievable fixed-rate (noiseless) source coding rate, and that this rate is equal to the sup-entropy rate of the random process. The Asymptotic Equipartition Property plays no role in the proof of this result, not only because it is not powerful enough to yield an approximation result in the sense of variational distance, but because the result holds for processes that are not necessarily ergodic or stationary. The proof uses a new technique we refer to as repetition. Some practical situations such as system simulation or the remote artificial generation of random processes such as speech sounds or image textures, suggest an important generalization of the foregoing setup: Given an input process and a channel, we want to approximate the resulting output process. However, this problem does not boil down to the previous setup when the approximation has to be accomplished by generating the input. We define the resolvability of a channel as the number of random bits per input sample required to achieve arbitrarily accurate approximation of the output statistics regardless of the actual input process. Intuitively, we can anticipate that the resolvability of a system will depend on how “noisy” it is. A coarse approximation of the input statistics whose generation requires Sergio Verdh Dept. Electrical Eng. Princeton University Princeton, NJ 08544 comparatively few bits will be good enough when the system is very noisy, because, then, the output cannot reflect any fine detail contained in the input distribution. Although the problem of approximation of output statistics involves no codes of any sort or the transmission/reproduction of information, its analysis and results turn out to be Shannon theoretic in nature. In fact, our main conclusion is that (for most channels) resolvability is equal to Shannon capacity. More concretely we show that the resolvability of an arbitrary channel is equal to the supremum of the input-output sup-information rate, and that this quantity coincides with the Shannon capacity if and only if the channel satisfies the strong converse. In addition to the abovementioned connections with the theories of source coding and channel coding, the approximation of output statistics is related to the problem of identification via channels introduced by Ahlswede and Dueck [11. Although a completely general direct identification coding theorem is known [U], its converse had been shown only in a so-called soft version in [l] and in the strong sense in 131, but always within the context of discrete memeoryless channels. Here, we show a general strong converse to the identification coding theorem which follows as a simple consequence of the achievability part of the resolvability theorem. The paper also investigates the effect of replacing the worst-case complexity measure by the average number of random bits required for approximation, as well as the replacement of variational distance by normahzed divergence. In the cases considered, the foregoing conclusions remain valid. We conclude with another result within the approximation theory of output statistics which formalizes a folk-theorem is channel coding: the output distribution due to any good channel code (a code with rate close to capacity and vanishing error probability) must approximate the output distribution due to the input that maximizes mutual information, and thus, achieves capacity. The journal version of this paper is to appear in [4].